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PROCEDURES USED IN DESIGNING 



A TRAINING PROGRAM 

IN EDUCATIONAL EVALUATION AND DEVELOPMENT 
General Procedu res 

The purpose of this first technical paper is to outline 
the procedures that were used in designing the trairing program 
in educational evaluation and development. In a second technical 
paper the description ard rationale of the consortium that 
evolved are presented. Appended to the second technical paper 
are working papers from several of the consortium units. A 
third technical paper in this series is concerned with the 
design which evolved from this planning activity, in effect, 
the proposed design for training educational evaluators and 
developers. Technical Paper 4 provides the budgetary information 
that amplify and in some respects clarify the program proposed 
in Technical Paper Number 3. 

It is of interest to note that in designing an original, 
novel training program for educational developers and evaluators, 
the project personnel have followed rather traditional task 
force procedures, after initial brainstorming sessions. The 
consortium units, now collectively titled the Colorado Center 
for Training in Educational Evaluation and Development (CCTEED) , 
proceeded through a series of both group and individual planning 
meetings, made decisions and counter decisions, and gradually 
approached their ultimate goal along a path that evidenced 
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frequent deviation from what would be considered a straight 
line. Curiously, although the staff of the Laboratory of 
Educational Research at the University of Colorado (the 
prime contractor) provided the day to day impetus to encourage 
focusing on the relevant issues, and attaining meaningful 
successive approximations, it was inputs from the consortium 
units and from "users" of educational evaluators and developers 
that provided the accumulation of data that in large measure 
shaped the allocation of resources in the final proposal. The 
responses of consortium units to various requests made of 
them for ideas and other information were invariably prompt, 
detailed, and conscientious ly stated. As such, these responses 
became valuable planning inputs. 

Sources of Data 

To a large extent, the procedures used in designing this 
training program in educational evaluation and development 
consisted of an elaborate collection of data from many sources, 
and a drafting of a tentative plan which was then screened, 
reviewed and critiqued by each of the consortium units. 

Their comments and opinions were incoiporated in the draft 
that was then submitted as the Preliminary Final Report. 

The sources of data consisted of: 



1) existing manpower studies; 

2) informational inputs from the U.S. Office of 



Education or its projects, 
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(a) the original RFP 70-12, 

(b) briefings with OE representatives, 

(c) The AERA Task Force on Research Training, 

(d) the Teaching Research Project to generate 
information to support planning for 
training in R, D, D, and E; 

3) group and individual meetings with consortium 
units ; 

4) working papers drafted by several of the 
consortium units; 

5) questionnaires sent to agencies nationwide and 
within the Rocky Mountain area; 

6) staff meetings at Colorado with frequent checks 
back to other colleagues and consortium units; and 

7) a projected conference on educational evaluation 
and development , 

Each of these sources of data is considered in more detail below 

1 . Existing Manpower Studies . 

Unfortunately, there is not a single source that has 
synthesized and incorporated the several manpower and related 
studies that were completed in the 1960 l s. {Although the AERA 
Task Force on Research Training is currently developing such 
a synthesis, it will not be completed until 1971.) Consequently 
it is necessary to study each of the various sources in turn 
and then attempt to draw some general conclusions therefrom. 
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In this connection, a series of different efforts were surveyed, 
namely the work of Bargar (1967) , Buswell, et al . ( 1966) , 

Clark and Hopkins (1969), Clark and Worthen (1967), Fattu (J 967}, 
Heiss (1966), National Center for Educational Research and 
Development (1969) , and Sieber (1966) . 

Generally the picture thus formulated was one portraying 
a gradually expanding research and development enterprise in 
the United States. Initially, the need for trained manpower 
seemed apparent in all the projections, but with the passage 
of time there seemed to be increasing doubt of the estimates 
of needed personnel primarily because of the beginning evidences 
of less support in this area by the federal government, that 
is, in the late 1960's. One would be led to believe from the 
various documents that much of the manpower need in R, D, D, ana E 
was met by a type of in-service training, often on an emergency 
basis, as research related personnel found new skills and new 
techniques demanded of them in their work. Despite the 
several training programs that have been initiated in the 
general area of research and research-related training, it 
was apparent that in none of the documents consulted was the 
opinion expressed that the number of new entries to the field 
who were being trained would be sufficient to meet the manpower 
needs of the educational research and development enterprise. 
Further, it was apparent that attention in most studies cited 
was focused on research training and often research training 
that might be designated as training 
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research rather than for decision or iented research. The 
studies do not consistently or expansively consider training 
in educational evaluation, development, and diffusion, although 
Bargar (1967) approximates tnis comprehensiveness. 

2 . U.S. Office of Education Sources. 

Initially, it should be mentioned that the request 
for proposals ( RFP 70-12) served as a valuable and powerful 
stimulus in terms of the thinking of the consortium units. 

That document made manifest the need in the research and 
research-related coramunity for*. training educational 
developers, diffusers, and evaluators; for retraining personnel 
already on the job, filling roles in these areas; and for 
training personnel below the level of the independent 
investigator (that is, support personnel). The RFP noted 
that the reason for the inclusion of a training title in the 
Elementary and Secondary Education Act of 1965 was to provide 
staff for the federal centers and laboratories, to assist in 
the staffing of the supplementary educational centers under 
Title ill of the act, to undertake the evaluation required 
of programs supported under Titles I and III of ESEA, as 
well as to provide personnel to work in other settings 
related to research. In a very real sense, it was suggested 
in the RFP that under* Title IV of ESEA, too much emphasis 
had been placed on the training of conclusion-oriented 
researchers, as distinguished from decision-oriented 
researchers. In no way was it suggested that there was not 
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a need for conclus ion-or iented educational researchers. 

The need for such individuals was and is apparent, and it 
remains great. What the RFP was stating, however, was that 
there was a requirement for new programs designed to train 
decision-oriented inquirers, and that this need certainly 
would not be met unless there was a major reallocation of 
resources and a fundamental rethinking of the processes by 
which educational evaluators and developers are trained. 

The RFP gave the persons working on this proposal an initial 
set to incorporate into the planning, and in the actual 
training, inputs received from what might be termed ” user 
organizations ' 1 (for example, public schools, state departments 
of education, laboratories and centers, etc.). 

The intent of the RFP and the evolving thinking of the 
U.S. Office of Education was conveyed in an additional 
source of data, namely briefings with personnel at the U.S. 
Office. Consequently, in late June a meeting was held <-t 
Washington for project directors. Individual conferences were 
held in early Septenibei in Washington with project directors, 
and other information was forthcoming via the R, D, D, and E 
manpower study being conducted at Teaching Research, as 
indicated below. In these briefings, it became apparent that 
there was a wide variation in the thinking of projec . design 
personnel as to the suitability and/or feasibility of training 
personnel in certain of these functional areas. More 
agreement existed on how to train evaluators than on how to 
train developers. There were even greater disparities on how 
q - train diffusion personnel. As will become apparent in 
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Technical Paper Number 3 in this series, the Colorado Center 
for Training Educational Evaluators and Developers plans to 
emphasize evaluation more than development, and excludes 
diffusion almost entirely. On this issue, and others, it 
was possible to interact extensively with personnel in the 
U.S. Office of Education during the design grant period of 
six months, a situation often not possible when proposals are 
being submitted for possible funding. 

Of substantial assistance to us as a source of data 
was the extensive series of technical papers that were forth- 
coming from the AERA task force project on training research 
personnel. The definitions of research, development, diffusion, 
and evaluation were helpful as well as the various skills 
listed for personnel holding jobs in these various functional 
areas. For example, the questionnaire discussed in section 
five below includes a list of skills of developers and 
evaluators that had been developed by the AERA task force. 

In addition, much data from the technical papers series 
seemed to suggest that there was a need, largely unmet, for 
educational evaluators and developers. Four sources of data 
provided the base for this conclusion. The first three were 
comparisons of the vacancies existing at the AERA placement 
service in 1968, 1969, and 1970, as compared with the number 
of applicants for positions, while the fourth was a survey 
of 58 employers of research and research- related personnel 
(see Technical Papers 3, 4, 5, 6, 7, 8, and 10). From these 
papers, it was clear that there was a great need for personnel 
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in educational evaluation and development; however, it also 
appeared that there was a trend in 1970 toward less willing- 
ness to hire new personnel in these areas and rather more 
stress on retraining personnel already employed by organiza- 
tions having responsibility in these areas. 

A second project which, like the AERA Task Force on 
Research Training, is also being supported with U.S. Office 
of Education funds, is the project housed at ^caching Research, 
a division of the Oregon State System of Higher Education. 

This project has as its main responsibility the generation 
of information that will support long-term manpower studies, 
and suppoit planning for training programs in educational 
R, D, D, and E. This project has been directly helpful 
in two regards in the formation of this proposal. First, 
via a cross-referencing conference in October, 1970, it was 
possible to interact with other designers of training 
programs, as well as with professors and others who had 
spent considerable time collecting, explaining, and writing 
about R, D, D, and E. The second major assistance from 
this project came by way of the conceptual papers that were 
developed for that project (the papers were presented by 
Br.igqs, Gideonese, Glass and Worthen, and Schalock and Sell). 

It should be noted that inputs from the last two sources 
of data mentioned, that is, the AERA task force project and 
the Teaching Research project, were readily incorporated 
into the thinking of the consortium 
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was possible because one of the consortium planners 
is directing the AERA task force project, while two 
of the Colorado planners wrote one of the conceptual 
papers for the Teaching Research project. In addition, 
three of the developers of this proposal were invited, 
in varying capacities, to attend the cross-referencing 
conference held in Seattle and did so. 
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3 . Meetings with Consortium Units. 

Several varied activities were carried out with 
consortium units, including both large group meetings, 
at which nearly all consortium units were present, and 
individual meetings, with each of the consortium units. 
The units that have been and are involved in 1 .e 
consortium are listed below: 

1) The University cf Colorado, Boulder, Denver, 
and Colorado Springs; 

2) The Center for Instructional Research and 
Curriculum Evaluation (CIRCE) , University 
of Illinois, Urbana, Illinois; 

3) The Biological Sciences Curriculum Study 
(BSCS) , Boulder, Colorado; 

4) The Earth Sciences Educational Program (ESEP) 
Boulder, Colorado? 

b) The High School Geography Project, Boulder, 
Colorado (terminated August 31, 1970); 

6) The Social Sciences Education Consortium 
(SSEC) , Bouluer, Colorado; 
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7) The S.rathwestern Cooperative Educational 

Laboratory (SWCEL) , Albuquerque, New Mexico; 

6) The Southwest Regional Laboratory for 

Educational Research and Development (SWRL) , 
Inglewood, California; 

9) The John F. Kennedy Child Development Center, 
University of Colorado Medical Center, 

Denver, Colorado? 

10) The Denver Public Schools, Denver, Colorado; 

LI) The Interstate Educational Resources Service 
Center, Salt Lake City, Utah; 

.12) The Northern Colorado Educational Board 

of Cooperative Services, Boulder, Colorado; 

13) The Colorado State Department of Education, 
Denver, Colorado; 

14} The Ford Foundation, New York, New York (and 
a Field Office for u. . leadership Development 
Program, Denver, Colorado) ; and 

15) The Office of Education Regional Office, 
Denver, Colorado* 

The meetings with these units provided valuable 
data for planning purposes. It was recommended to the 
units that they emphasize strongly those things which 
they felt were in their organization's best interest, 
hoping that by this procedure bonds between the consortium 
units might be more direct and lasting. To this end, the 
source of data mentioned below was undertaken. 
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4 . Working Papers of Consortium Units . 

Consortium units were asked to develop independent 
working papers early in the course of the project. Those 
units that were unable to dc j "> (primarily because of 
time constraints) were contacted individually, so that 
their specific reactions to the issues considered in 
the working papers could be incorporated into the project 
planning. The outline that was used as a guide for the 
development of the working papers is included as Appendix A 
Of this technical paper. The working papers actually 
produced have been included as appendices in Technical 
Paper Number 2. The reader is referred to that source 
for further elaboration of the issues that they address. 
Suffice it to say that it was from the content of these 
working papers that the design project personnel began 
to move toward "intensive training institutes" described 
in Technical Paper Number 3. Addi tionally , it was felt 
that the working papers further confirmed the belief 
that many organizations were finding it difficult to 
add personnel , because of financial constraints, and 
were more concerned with the retraining of existing 
personnel in new skill areas. 

5 . Questionnaire on Educational Evaluation and Development. 

After considerable feedback had been received, 
the general types of training and levels of training 
that would be offered in this training program began to 
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emerge. However , It was felt necessary to survey a 
wider base of organizations in order to determine more 
accurately the specific demands for different type3 
of training programs by personnel already employed in 
educational development and evaluation. To this end, 
a questionnaire was designed and mailed to a sample of 
about 30 0 different organisations. The questionnaire 
and the cover letter that accompanied it are included 
as Appendix B in this technical paper. The interested 
reader is referred to that appendix to determine the 
exact questions that were asked. It should be indicated 
that the data forthcoming from the questionnaire are 
reported in Technical Paper Number 3 in this series. 

The sample used was deliberately selective in cer- 
1 ain respects in that the sample was selected within 
predetermined categories. Questionnaires were sent to 
all R & D centers, all regional laboratories, all 
ERIC organi za tion s, al 1 consortium units, the 30 largest 
school districts in the United States, and to selected 
research projects, agencies, and organi zations who 
were thought to have personnel needs in evaluation and 
development. Additionally, via two consortium units, 
school districts of a smaller size were sampled; i.e., 
all seven school districts in the Northern Colorado 
Educational Board of Cooperative Cervices Organization 
were included as were a random one-third of the moderate 
size school districts in the eight states served by the 
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Interstate Educational Resources Service Center in 
Salt Lake City (Arizona, Colorado, Idaho, Montana, 

Nevada, New Mexico, Utah, and Wyoming). A moderate 
size school district was interpreted as one having 
between 2500 and 10,000 students. Districts under 
this size were not sampled because it was felt unlikely 
that they have personnel currently engaged in evaluation 
or development to any large extent. 

6 . Staff Meetin gs . 

Frequent meetings were held by the four staff 
members at the University of Colorado who were principally 
charged with the responsibility to develop and design 
the training program. These meetings were called on an 
ad hoc basis, and were held at least once weekly. In 
intervening periods perceptions, plans, and contemplated 
procedures were checked with colleagues both at the 
university and nationally, as well as with consortium 
units. The luxury of having six months to plan such 
a training program was unusual, but beneficial for a 
number of reasons. One primary reason is indicated in 
the closing paragraph of this technical paper? that is, 
it is essential that the planning period be long enough 
to allow data from many sources to have an impact upon 
and influence the resultant design. 
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7 . The Evaluation an d Development Blow Your Mind Conference. 

This conference v/as held on December 9-10 , 1970, 

Three consultants provided most of the input: Dr, Arthur 

Lumsdaine f Professor of Psychology, University of Washington; 
Dr* Sam Messick, Educational Testing Service; and Dr. Sam 
Sieber, Bureau of Applied Social Research, Columbia University* 
The team of consultants assembled for two primary purposes* 
First, they investigated and critically evaluated the 
capacity ol the existing structure at Colorado to adminis- 
tratively and ins tructionally support an evaluation and 
development training program. Second, they were encouraged 
to free associate and otherwise brainstorm on the essential 
elements and other ramifications involved in training 
educational evaluators and developers. An abbreviated 
summary on the conference will be appended to this final 
report as Appendix C. Since the final report will be 
mailed about the time that the conference concludes, 

Appendix C will not be included but will be forwarded as 
soon as available. 

The title of the conference, although somewhat 
prostituted by use of current vernacular, resulted from 
the intention to encourage and in every way 'acilitate 
the creation of "way-out" training ideas and concepts. 

It is felt that many such ideas, after appropriate mellowing 
during an intervening time period, conceivably could 
become feasible and valuable contributions to the training 
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It is interp.sUng to note the evolution that has 
occurred, over the period that the program has been 
designed and planned in the thinking of both the 
personnel involved at the University of Colorado and 
of those associated with the consortium units. When 
the original proposal was drafted in May, 1970, it 
served, in effect, as a type of baseline for our thinking 
at that point in time. Subsequent ly , the August 1 
progress report, and then the outline and oral report 
to the U.S. Office of Education in early September, 1970, 
served as marking places denoting shifts in orientation, 
in assumptions, in feasibility, and in concepts of the 
program being designed. Likewise, the Preliminary 
Final Heport served as a type of current perspective on 
the same issues. It is likewise interesting to note the 
pronounced differences that exist between the plan now 
conceptualized (presented primarily in Technical Paper 
Number 3) as compared to the type of training program 
that probably would have emerged had that been the initial 
charge in May, 1970. The resultant changes, primarily in 
the direction of increased effectiveness and enhanced 
feasibility, almost certainly would not have evolved 
without the six month design grant. 
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Evaluation, Development and Diffusion 
Training Design Project 

Laboratory of Educational Pesearch 

University of Colorado 28 August 1970 

Suggested Format for Working Papers 
of Consortium Constituent Agencies 



I. In your organization, what personnel needs exist for persons 

trained in evaluation, development, and diffusion? 

A. How many personnel, and at what level (Bachelors, Masters, 
Ph.D.), are presently employed in evaluation? in develop- 
ment? in diffusion? 

B. How many additional personnel, and at what level, are need- 
ed in evaluation? in development? in diffusion? 

C. If your organization's annual budget were increased by 
$100,000, how many additional persons, and at what level, 
would be hired in evaluation? in development? in diffu- 
sion? 

D. What specific skills {e.g., in design, research methodol- 
ogy, statistical analysis, objectives analysis, measure- 
ment, instrumentation, change strategies, research, etc.) 
would be needed by such personnel working for your organi- 
zation? 



II . 




Given the personnel presently employed by your organization 

in evaluation, development, and diffusion, what needs have 

they for additional training? 

A. Further training is needed in what specific skills (e.g., 
in design, research methodology, statistical analysis, 
objectives analysis, measurement, survey research, instru- 
mentation, change strategies, etc.)? 

B. What periods of time (e.g., two weeks, six weeks, one 
semester, one year, etc.) for such training would be 
preferable? What periods of training would be feasible? 

C. Would persons so trained probably return to your organiza- 
tion in their former position, in a redefined position, 

or in a new position? 

D. How much financial support do you feel would be necessary 
to interest the person in the preferred training periods 
described above? 

E. Would your organization give employees a year's leave of 
absence for training? Would it grant a leave of absence 
for shorter time periods iplease be specific on duration 
on leave of absence that would be authorized)? 
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F. 

G. 

H. 



Would your org 
to supplement 
stipend is $75 
nual stipend i 
Can you think 
who would avai 
in evaluation? 
Would the poss 
a result of su 
for persons i 



anization provide th 
a trainee’s stipend 
plus $15 per depend 
s $2,400, plus $500 
of persons currently 
1 themselves of such 
in development? i 
ibility of earning a 
ch training serve a 
n your organization? 



e monies necessary 
(the current weekly 
ent; the current an- 
per dependent)? 
in your organization 
training? How many 
n diffusion? 

Master’s degree as 
s a strong incentive 



III. What specific training program components (courses, intern- 
ship experiences, etc.) do you feel would be best to bring 
about the skills described in item I.D above (primarily 
in a training program for bringing new personnel into the 
field) ? 



IV. What specific training program components (courses, intern- 
ship experiences, etc.) do you feel would be most important 
to bring about the skills described in item II. A above (pri- 
marily in a training program for persons already employed 
by your organization)? 

V. How do you envision that your organization might be best 
used, both as an instructional site and as an internship 
site for training evaluators? developers? diffusion 
agents? Are there individuals presently employed by your 
organization who (a) could supervise an internship in your 
agency in evaluation, development, or diffusion, and (b) 
would be willing to do so? 
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TECHNICAL PAPER NUMBER 1 
APPENDIX B 
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UNIVERSITY OF COLORADO 



0OUL.rJcR, COLORADO 00302 



.AlOmlCjKr a* |OUf »TiON*L 



As you may know, under Title IV of the Elementary and 
Secondary Education Act, the USDS has funded twelve agencies 
to develop nev; train inu models and programs for personnel in 
educati unal research, development , diffusion, end evaluation. 

A consortium in and around Colorado received one cf the grants, 
and has subsequently focused its design efforts on development 
and evaluation. 

Enclosed is a questionnaire (one side dealing with 
evaluation, the other with development) and a pre- addressed 
envelope. Given your leadership role in your organization, 
your responses would be extremely helpful to us in our plan- 
ning. Xn some cases, you may want particular employees or 
associates working with you to answer certain of the questions 
because of their involvement with evaluation and development. 

I realize that you are bombarded with similar requests, yet, 
as always, it is hoped that you can respond at once, possibly 
even returning the completed questionnaire in today r s mail* 

An effort has been made to minimize the time required to 
complete the instrument; fifeern to twenty minutes appears 
to be about average. 

Thank you in advance for providing this professional 
response. Your inpuc, and the inputs of other 01‘ganization 
heads, will be instrumental in designing certain features of 
the training program. 

S incere ly , 

William L. Goodwin 

Design Project Director 
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Assuming a 20 % iucr ease in your annual budget, how many evaluators would 
your organization hire next year, and at what level? 

at the BS level? at the IIS level; at the PhD level. 

Assuming no increase in your present annual budget, how many e valuators 
would your organization hire next year, and at what level, either as 
replacements or as .persons filling nev; slots resulting from reallocation 
Of resources? 

at the 3S level; __ at the MS level; at the PhD level. 

If high quality, short-term training institutes were held in evaluati on , 
how many staff in your organization would be permitted to take how much 
training? (Assume that your staff continue to receive their regular 
salary from your organization, while a training stipend from external 
sources defrays their costs of living away from home.) 

Please fill Dither could attend one 2 -week institute annually, 

in all three or could attend two 2 -week institutes annually; 

blanks. or could attend three 2-week institutes annually. 

Assuming that you were hiring a new eval uator , rank order belov? the impc 
tance of his having each of the following skills, from the perspective 
of your organization. (Rank all the skills; rank the most important 
skill as 1, etc.) 

If the short-term training institute was held in evaluation . , rank order 
belov; the importance of training in each of the following evaluation 
skills, from the perspective of your organization. (Rank all the skill?, 
rank the most important skill as 1, etc.) 

£7 

Identifying goals of the program to be evaluated, and the goals' 

social relevance and implicit values. 

_ Helping system personnel develop objectives to satisfy needs or 

1 solve problems, and establishing priorities among these objective. 

] 

| Translating broad objectives into specific observable objectives* 

Identifying standards or norms decision-makers will us e in inter 

preting data. 

j Establishing an appropriate evaluation design for the program 

| involved. 

i Monitoring the program to detect deviations from the design cr 

specified procedures. 

Selecting or developing and using valid measurement techniques 
: to yield information on outcomes, 

Employing appropriate techniques of data analysis. 

Making recommendations as the result of evaluation. 

Writing the evaluation report, or otherwise reporting the result 



Other (Specify) 
Other (Specify) 
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Position Organisation 

1. What is the present composition of your organization’s professional staf 

at the 3S level; at the MS level/ at the PhD level. 

2. How much of your professional staff’s time is spent primarily on activit 
that involve eva luation ? (Express in full-time equivalents; c.g., if 
three staff each spend half-time in evaluation, the full-time equivalent 
is 1.5 FTE.) 

FTE at the BS leveJ ; FT 3 at the KG level, FTC at the PhD leve 

3. Assuming a 20% increase in your annual budget, how many evaluators would 
your organization hire next year, and at what level? 

at the BS level; at the MS level? at the PhD level. 

4. Assuming no increase in your present annual budget, how many e valuators 
would your organization hire next year, and at what level, either as 
replacements or as .persons filling new slots resulting from reallocation 
of resources? 

at the *3S level? at the MS level; at the PhD level. 

5. If high quality, short-term training institutes were held in evaluat ion , 
how many staff in your organization would be permitted to take how much 
training? (Assume that your staff continue to receive their regular 
salary from your organization, while a training stipend from external 
sources defrays their costs of living a 1 . /ay from home.) 

Please fill Either could attend one 2 -week institute annually, 

in all three or could attend two 2 -week institutes annually; 

blanks. or could attend three 2 -week institutes annually. 

6. Assuming that you were hiring a nev? evalua tor , rank order below the impc 
tance of his having each of the following skills, from the perspective 
of your organization. (Rank all the skills; rank the most important 
skill as 1, etc.) 

7. If the short-term training institute was held in evaluat ion , rank order 
below the importance of training in each of the follov/ing evaluation 
skills, from the perspective of your organization. (Rank all the skill? 
rank the most important skill as 1, etc.) 

Identifying goals of the program to be evaluated, and the goals' 
social relevance and implicit values. 

Helping system personnel develop objectives to satisfy needs or 
solve problems, and establishing priorities among these objective 

Translating broad objectives into specific observable objectives. 

Identifying standards or norms decision-makers will use in inter 
preting data. 

Establishing an appropriate evaluation design for the program 
involved. 
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of resources? 

at the BS level; ^ at the MS level; at the PhD level. 

1. If high quality, short-term training institutes were held in development , 
hov; many staff in your organization would be permitted to take hov; much 
training? (Assume that your staff continue to receive their regular 
salary from your organization, while a training stipend from external 
sources defrays their costs of living away from home.) 

Please fill Either could attend one 2-week institute annually, 

in all three or could attend two 2-v/eek institutes annually; 

blanks. or could attend three 2-v;eek institutes annually. 

2. Assuming that you v;ere hiring a nev; de veloper , rank order below the 
importance of his having each of the - following skills, from the perspecti" 
of your organization. (Rank all the skills; rank the most important skil. 
as 1, etc.) 

j. If the short-term training institute was held in deve l opmen t , rank order 
below the importance of training in each of the following development 
skills, from the perspective of your organization. (Rank all the skills; 
rank the most important skill as 1, etc.) 

12 v 1 3 

Interpreting, evaluating, and synthesizing relevant literature, 

and drawing on research results in planning developmental 
f activities. 

Developing instructional systems, their elements, and interrela- 
tions among these elements. 

___ __ Specifying desired performance objectives, and establishing 

standards for judging attainment of objectives. 

Choosing appropriate instructional and media techniques in 

developing educational products and/or processes. 

Determining appropriate sequence? of topics in instruction. 

Developing products based on effective oral and wri tten forms 

of instructional communications. 

_ Directing the work of production personnel. 

Selecting or developing and using appropriate techniques for 

measuring outcomes. 

Designing and managing laboratory tests of developed techniques 

and materials, and also field tryouts and tests. 

Reporting evaluation of outcomes and specifying requirements 

for revision based upon outcome evaluation. 

Other (Specify) 

_ Other (Specify) ■ 
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3. How much of your professional staff's time is spent primarily on activit 
that involve development ? (As in £2, express in f ull- time-equivalen ts . ) 

FTE at the 3S level; __FTS at the MS level; FTE at the PhD lev 

), Assuming a 202 increase in. your annual budget, how many developers v/ould 
your organization hire next year, and at what level? 

at the BS level? at the MS level; at the PhD level# 

)# Assuming no increase in your present annual budget, hov; many developer s 
would your organization hire next year, and at what level, either as 
replacements or as persons filling new slots resulting from reallocation 
of resources? 

at the BS level; at the MS level? at the PhD level# 

1. If high quality, short-term training institutes were held in development , 
how many staff in your organization would be permitted to take hov; much 
training? (Assume that your staff continue to receive their regular 
salary from your organization, while a training stipend from external 
sources defrays their costs of living away from home 0 

Either could attend one 2-week institute annually, 

or could attend two 2-week institutes annually; 

or could attend three 2-week institutes annually, 

2. Assuming that you were hiring a new de veloper , rank order below the 
importance of hi s having each of the" following skills, from the perspecti 
of your ora-ni zation * (Rank all the skills? ran}; the most important skil 
as 1 , etc. ) 

3. If the short-term training institute was held in developmen t , rank order 
below the importance of training in each of the following development 
skills, from the perspective of your organization. (Rank all the skills; 
rank the most important skill as 1, etc.) 

12 £13 

Interpreting, evaluating, and synthesizing relevant literature, - 
and drawing on research results in planning developmental 
activi tics • 

Developing ins truct ional systems, their elements, and interrela- 
tions among these elements. 

Specifying desired performance objectives, and establishing 
standards for judging attainment of objectives. 

Choosing appropriate instructional and media techniques in 
developing educational products and/or processes# 

Determining appropriate sequences of topics in instruction. 

Developing products based on effective oral and written forms 
of instructional communications . 

Directing the work of production personnel. 

Selecting or developing and using appropriate techniques for 
measuring outcomes. 




Please fill 
in all three 
blanks . 



